Information processing apparatus and information processing method

ABSTRACT

An information processing apparatus may include a detection unit and a switching unit. The detection unit detects an amount of change in a position of an object of interest per a predetermined time period. The switching unit switches between a first mode for determining a first operation position on a display surface based on the position and direction of an object of interest and a second mode for determining a second operation position on the display surface based on a position where the object of interest is in contact with the display surface using the detected amount of change.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique for switching an inputmode.

2. Description of the Related Art

Conventionally, there has been known a technique in which a touch inputoperation to a display screen points a predetermined position on thedisplay screen. Furthermore, there has also been known a technique inwhich pointing input operation to a display screen with a finger pointsa predetermined position on the display screen.

It is difficult for a pointing input operation to accurately point at aremote object due to the influence of hands movement. Therefore, it isdesirable to point at a rough position on the display screen by thepointing input operation and further to perform a touch input operation,thereby pointing at a fine position on the display screen.

However, to perform switching between modes of receiving the pointinginput operation and the touch input operation, it has been required torecognize a speech of switching instructions and receive a buttonoperation for switching instructions.

SUMMARY OF THE INVENTION

The present invention is directed to providing switching an input modebased on the movement of a finger itself (an object of interest)performing the pointing input operation.

According to an aspect of the present invention, an informationprocessing apparatus may include a detection unit and a switching unit.The detection unit detects an amount of change in a position of anobject of interest per a predetermined time period. The switching unitswitches between a first mode for determining a first operation positionon a display surface based on the position and direction of an object ofinterest and a second mode for determining a second operation positionon the display surface based on a position where the object of interestis in contact with the display surface using the detected amount ofchange.

Further features and aspects of the present invention will becomeapparent from the following detailed description of exemplaryembodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate exemplary embodiments, features,and aspects of the invention and, together with the description, serveto explain the principles of the invention.

FIGS. 1A and 1B are schematic diagrams illustrating the installation ofa system.

FIGS. 2A and 2B illustrate a functional block diagram and a hardwareconfiguration of an information processing apparatus.

FIG. 3 illustrates a flow chart indicating the process for switching aninput mode.

FIGS. 4A to 4C are flow charts indicating detail processes in steps S402and S403.

FIG. 5 is a schematic diagram illustrating that an input mode isswitched.

FIG. 6 is a flow chart indicating detail process in step S404.

FIG. 7 is a schematic diagram illustrating that an input mode isswitched.

FIGS. 8A to 8C are tables illustrating results in step S502.

FIG. 9 is a graph illustrating results in step S502.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments, features, and aspects of the inventionwill be described in detail below with reference to the drawings.

FIG. 1 is a schematic diagram illustrating the installation of a system(an information processing system) according to a first exemplaryembodiment. The system includes a projector 101, a camera 102, adigitizer 103, and an information processing apparatus 104. In thepresent exemplary embodiment, the system in which the above devicescollaborate with one another realizes an embodiment, however, a singleinformation processing apparatus having functions corresponding to theabove devices may realize an embodiment.

The projector 101 projects a predetermined image onto the digitizer 103according to the image data output from the information processingapparatus 104. The camera 102 is arranged opposed to the digitizer 103and captures an object of interest such as the finger of a user. Thecamera 102 is a range image sensor for outputting a range image. Therange image refers to an image whose each pixel has a pixel valueaccording to a distance between the camera 102 and an object of interestin a space.

For example, a time-of-flight (TOF) measurement method in which infraredrays are emitted and time until light reflected by an object reaches ismeasured is applied to a method for measuring a distance. A plurality ofrange images output at points of time different from each other iscompared to allow identifying the moving distance and the movingdirection of the object of interest. The camera 102 has a function torecognize an object with a predetermined shape and is capable ofrecognizing the shape of user's hand (such as a fingertip, the base of afinger, and others).

In the present exemplary embodiment, various shapes of a human being anda hand are stored in a storage device (RAM) as a model to realize theabove recognition functions. The information processing apparatus 104may also be provided with a function to recognize an object with apredetermined shape.

The digitizer 103 detects the position of an object brought into contactwith the digitizer 103. The digitizer 103 functions as a display surfacefor the image projected by the projector 101. The information processingapparatus 104 acquires the range image from the camera 102 andinformation about contact position from the digitizer 103, executes theaftermentioned processing, and outputs predetermined image data to theprojector 101.

In the present exemplary embodiment, the digitizer 103 is rectangularand 190 cm in length (X-axis direction) and 150 cm in width (Y-axisdirection). For the three-dimensional space over the digitizer 103, apoint P1 corresponding to the corner of the digitizer 103 is set to anoriginal point (0, 0, 0), the width direction is set to a Y-axisdirection, the length direction is set to an X-axis direction, and theheight direction is set to a Z-axis direction (passing the originalpoint and being orthogonal to the XY plane). In the present exemplaryembodiment, points P2 and P3 corresponding to other corners of thedigitizer 103 are set to (190, 0, 0) and (0, 150, 0). The coordinate isin cm.

FIG. 2A illustrates a functional block diagram of the informationprocessing apparatus 104. The information processing apparatus 104includes a pointing input processing unit 201, a touch input processingunit 202, a mode management unit 203, a display control unit 204, and aremote control unit 205. The information processing apparatus 104 isconnected to the projector 101, the camera 102, and the digitizer 103.The pointing input processing unit 201 is comprised of a CPU and others,obtains the position of an object of interest (a finger in the presentexemplary embodiment) and the direction of the object of interest (thedirection of a finger in the present exemplary embodiment) from therange image output by the camera 102, and identifies a predeterminedposition over the digitizer 103 (display surface).

In the three-dimensional spatial coordinate system illustrated in FIG.5, for example, it is assumed that the position coordinate P4 of afingertip is obtained as (90, 120, 50) by the pointing input processingunit 201 and the position coordinate P5 of the bottom of a finger isobtained as (93, 130, 53). In this case, the intersection (60, 20, 0) ofthe straight line connecting the position coordinate of the fingertipwith the position coordinate of the base of the finger and the facecorresponding to the digitizer 103 (display surface) is identified as apointing position (an operation position).

The pointing input processing unit 201 manages the history ofinformation indicating pointing input. In the present exemplaryembodiment, although the pointing input is performed using a finger, thepointing input may be performed using a pen tip. Pointing may beperformed using a pointing device such as laser pointer. The touch inputprocessing unit 202 is comprised of a CPU and recognizes contact withthe digitizer 103. The touch input processing unit 202 manages thehistory of information indicating touch input.

The mode management unit 203 is comprised of a CPU and has a function toswitch between the pointing input mode and touch input mode. In thepointing input mode, the position identified by the pointing inputprocessing unit 201 is pointed. In the touch input mode, the position atwhich contact with the digitizer 103 is detected as the positionidentified by the touch input processing unit 202 is pointed. Thedisplay control unit 204 is comprised of a CPU and an output I/F andoutputs a predetermined image data to the projector 101. The remotecontrol unit 205 is comprised of a CPU and a RAM and executes controlfor remotely controlling the pointing position described later.

FIG. 2B illustrates a hardware configuration of the informationprocessing apparatus 104. The information processing apparatus 104includes a central processing unit (CPU) 301, a random access memory(RAM) 302, a read only memory (ROM) 303, an input interface (I/F) 304,an input I/F 305, and an output I/F 306. The information processingapparatus 104 is connected to the projector 101, the camera 102, and thedigitizer 103.

The CPU 301 develops the program stored in the ROM 303 in the RAM 302and executes the program, realizing each function of the informationprocessing apparatus 104. The RAM 302 provides a storage area requiredfor the operation of the above program. The ROM 303 stores program anddata for realizing the operation process of the above program. Theinterface I/F 304 acquires range image from the camera 102. The inputI/F 305 acquires information indicating contact position from thedigitizer 103. The output I/F 306 outputs predetermined image data tothe projector 101. The above devices function in collaboration with oneanother via a system bus.

FIG. 3 illustrates a flow chart indicating the process for switching theinput mode. In step S401, the mode management unit 203 determineswhether the input mode is the pointing or the touch input mode. In stepS401, if contact with the digitizer 103 is detected, the mode managementunit 203 determines that the input mode is the touch input mode andexecutes the processing in step S405. On the other hand, if contact withthe digitizer 103 is not detected, the mode management unit 203determines that the input mode is the pointing input mode and executesthe processing in step S402. In step S402, the pointing input processingunit 201 captures the finger of the user as the object of interest andidentifies the pointing position on the display surface (the digitizer103) based on the position and direction of the object of interest (thefinger).

In step S403, the pointing input processing unit 201 detects whether thefinger is brought down based on the range images output at points oftime different from each other. More specifically, in step S403, thepointing input processing unit 201 determines whether the amount ofmovement (amount of change) of the fingertip in the directionperpendicular to the digitizer 103 (display surface) per a predeterminedtime period (280 msec, for example) is equal to or more than apredetermined distance (13 cm, for example) in the direction close tothe display surface. If the pointing input processing unit 201 detectsthat the finger is brought down (YES in step S403), the processing instep S404 is executed. If the pointing input processing unit 201 detectsthat the finger is not brought down (NO in step S403), the processing instep S402 is executed.

In step S404, the mode management unit 203 executes the processing forswitching the input mode from the pointing input mode to the touch inputmode. In step S408, the touch input processing unit 202 performs thetouch input processing.

In step S405, the touch input processing unit 202 identifies thepointing position based on the contact position of the finger on thedisplay surface (the digitizer 103). Specifically, for example, theposition where the digitizer 103 is brought into contact with thefingertip is the pointing position. In step S406, the pointing inputprocessing unit 201 detects whether the finger is brought up. Morespecifically, in step S406, the pointing input processing unit 201determines whether the amount of movement (amount of change) of thefingertip in the direction perpendicular to the digitizer 103 (displaysurface) per a predetermined time interval (280 msec, for example) isequal to or more than a predetermined distance (13 cm or more, forexample) in the direction away from the display surface.

If the pointing input processing unit 201 detects that the finger isbrought up (YES in step S406), the processing in step S407 is executed.If the pointing input processing unit 201 detects that the finger is notbrought up (NO in step S406), the processing in step S405 is executed.In step S407, the mode management unit 203 performs switching from thetouch input mode to the pointing input mode. In step S409, the pointinginput processing unit 201 performs the pointing input processing. Inother words, the processing in steps S401 to S407 allows switchingbetween the pointing input mode and the touch input mode by theoperation with a single finger.

The determination methods in steps S403 and S406 are not limited to theabove ones. For example, instead of the above processing in step S403,the pointing input processing unit 201 acquires the gradient (s1) of thestraight line obtained by the linear approximation of the fingertipposition detected from the reference point of time to the current pointof time. Then, the pointing input processing unit 201 acquires thegradient (s2) of the straight line obtained by the linear approximationof the fingertip position detected from the reference point of time tothe point of time retrospective by a predetermined amount of time andcompares both gradients. If the value of s2/s1 is equal to or greaterthan a predetermined threshold, the pointing input processing unit 201may determine that the finger is brought down. Furthermore, instead ofthe above processing in step S406, the pointing input processing unit201 acquires the gradient of the straight line obtained by the linearapproximation of the fingertip position detected at from the referencepoint of time to the current point of time. Then, the pointing inputprocessing unit 201 acquires the gradient of the straight line obtainedby the linear approximation of the fingertip position detected at fromthe reference point of time to the point of time retrospective by apredetermined amount of time and compares both gradients. Thereby, thepointing input processing unit 201 may detect whether the finger isbrought up.

Processing methods (S501 to S505) in step S402 are described in detailbelow using FIG. 4A. FIG. 4A is a flow chart illustrating the procedureof internal processing in step S402. In step S501, the pointing inputprocessing unit 201 detects the point P4 of the fingertip (index finger)of a hand 105 and the point P5 of the base thereof from the imagecaptured by the camera 102. In step S502, the pointing input processingunit 201 recognizes the distance and direction from the original pointwith respect to the point P4 of the fingertip from the image captured bythe camera 102. The pointing input processing unit 201 identifies thecoordinates of the point P4 of the fingertip in the three-dimensionalspace. The pointing input processing unit 201 acquires also temporalinformation from the image captured by the camera 102 to associate thetemporal information with the coordinates in the three-dimensionalspace. The pointing input processing unit 201 continues to acquiretemporal information and coordinates in the three-dimensional space at apredetermined time interval and stores the acquired temporal informationand coordinates in the three-dimensional space in a storage unit.

FIG. 8 is a table illustrating results obtained from the processing instep S502. The table lists the X-coordinate X1, the Y-coordinate Y1, andthe Z-coordinate Z1 of the point P4 with respect to time T (units inmsec). FIG. 9 is a graph of the information illustrated in FIG. 8. Thehorizontal axis in FIG. 9 represents time T and the vertical axisrepresents the X-coordinate X1, the Y-coordinate Y1, and theZ-coordinate Z1. The point of time when the user protrudes the indexfinger of the hand 105 and starts the pointing input is set to T=0.Moment at which the index finger is protruded is detected from thecaptured image.

In step S503, the pointing input processing unit 201 identifies thecoordinates of the point P5 of the base of a finger in thethree-dimensional space in the manner similar to that in step S502. Thepointing input processing unit 201 acquires also temporal information toassociate the temporal information with the coordinates in thethree-dimensional space in the manner similar to that in step S502. Instep S504, the pointing input processing unit 201 takes the intersectionP6 of the straight line connecting the point P4 to the point P5 and theXY plane as a pointing position. The equation of the straight lineconnecting the point P4 to the point P5 is obtained and the X-coordinateX3 and the Y-coordinate Y3 at which the Z-coordinate becomes 0 areobtained. In step S505, the projector 101 being a display unit displaysa cursor on the point P6. The pointing input processing unit 201continues to identify the pointing position during the period of thepointing input mode and a cursor is displayed during the period.

The pointing position refers to a position at which the user points bytouching or indicating with the finger. The cursor is a symboldisplaying the pointing position on the display. A cursor position is aposition of the cursor before a transition from the pointing mode to thetouching mode and the pointing position before the transition. After thetransition, the position on this side at which the user points isreflected in the movement of a remote cursor position.

Processing methods (S601 to S602) in step S403 are described in detailbelow. FIG. 4B is a flow chart describing the procedure of internalprocessing in step S403. In step S601, the pointing input processingunit 201 detects a changing point from the relationship between the timeT and the amount of change in the fingertip stored in the storage unit.If the pointing input processing unit 201 detects the predeterminedamount of change in a fingertip during a predetermined time period (YESin step S601), the processing in step S602 is performed. If the pointinginput processing unit 201 does not detect a changing point (NO in stepS601), a series of the processing is ended.

In the present exemplary embodiment, the processing in step S602 isperformed when an interval during which the Z-coordinate Z1 is decreasedby a predetermined distance (13 cm in the present exemplary embodiment)or longer within from the point of time when measurement is started(T=0) to a predetermined time period (280 msec in the present exemplaryembodiment) or shorter is detected for the first time. As shown in FIG.8B, since the amount of change in the Z-coordinate Z1 is decreased by13.02192 in the range between T1 (T=1440 msec) and T2 (T=1680 msec), theprocessing in step S602 is performed. The point of time corresponding tothe point of time T1 is taken as a changing point. In step S602, thecursor is stopped at the position where the changing point is detectedin step S601.

In a case where the processing in step S602 is not performed, thepointing position is moved in line with the user's finger being broughtdown. The processing in step S602 is performed to achieve an effectcapable of switching the pointing input mode to the touch input modewith the user-intended pointing position maintained. The processingmethod in step S406 is described below. As is the case with theprocessing in step S501, the coordinate of the point P4 of fingertip(index finger) of the hand 105 in the three-dimensional space isidentified from the image captured by the camera 102. As is the casewith the processing in step S404, a changing point is detected from therelationship between the time T and the amount of change in thefingertip. If a predetermined amount of change in a fingertip isdetected during a predetermined time period, the processing in step S408is performed. If the changing point is not detected, the processing instep S601 is continued.

In the present exemplary embodiment, the processing in step S407 isperformed when an interval during which the Z-coordinate Z1 is increasedby 13 cm or longer within from the point of time when measurement isstarted (T=0) to 280 msec or shorter is detected for the first time.Detailed description using data is made in steps S403 and S404, so thatthe description of step S407 is omitted herein.

The processing in step S408 is described in detail below. FIG. 6 is aflow chart illustrating the procedure of internal processing in stepS408. In FIG. 6, “pointing input mode” denotes time at which thepointing position is identified in step S504 and the point of time T1.“switching” denotes time at which the pointing position is identified instep S602 and the point of time T2.

In step S801, the touch input processing unit 202 determines whether thepointing position identified at the time of the pointing input mode(step S504) lies on an image object. The image object denotes a partialimage movable on the screen among partial images displayed on thescreen, for example, a photograph, icon, and document. If the pointingposition lies on an image object A at the time of the pointing inputmode (YES in step S801), as illustrated in FIG. 7, the processing instep S802 is performed.

If the pointing position does not lie on the image object A at the timeof the pointing input mode (NO in step S801), the processing in stepS807 is performed. In step S802, the touch input processing unit 202determines whether the hand 105 (subjected to the pointing inputprocessing) touches another image object at the time of switching thepointing input mode to the touch input mode in step S404. If the hand105 touches a position different from the image object (NO in stepS802), the processing in step S804 is performed. Further if the hand 105touches the image object B at the time of switching (YES in step S802)as illustrated in FIG. 7, for example, the processing in step S803 isperformed. In step S803, the image object A is replaced with the imageobject B and the touch input processing is performed in step S810.

In step S804, the touch input processing unit 202 determines the numberof the fingers that touch the points in step S802. If the number oftouch points is one, the processing in step S805 is performed. If thenumber of touch points is two, the processing in step S806 is performed.In step S805, the pointing position identified immediate before thepointing input mode is operated by touch input in the touch input mode.In step S806, a menu related to the image object A is displayed in thetouch input mode. For example, touch input is performed in the positivedirection of the X-axis to display a menu related to the change of animage size. In step S811, the touch input processing is performed. Instep S807, the touch input processing unit 202 determines whether thehand 105 touches the image object when the pointing input mode isswitched to the touch input mode in step S404. If the hand 105 touchesthe image object B, for example, (YES in step S807), the processing instep S808 is performed. If the hand 105 does not touch the image objectB (NO in step S807), the processing in step S809 is performed. In stepS808, the image object B is moved to the pointing position identified inthe pointing input mode. In step S811, the touch input processing isperformed.

In steps S805 and S809, the pointing position identified in the pointinginput mode is operated by touch input in the touch input mode. It isdifficult to perform operation such as accurate pointing at a remoteimage object or movement by the pointing input due to the influence ofhands movement. Rough pointing at an image is performed by the pointinginput and then detailed pointing operation is performed by touchoperation on a table. This achieves an effect capable of accuratelyoperating a remote image object.

At the time of performing a remote operation in steps S805 and S809, theimage object around the pointing position identified in the pointinginput mode may be copied into a touch input area to allow performing theremote operation. Thereby, the user can perform touch input, whilewatching the hand, into an image object in an area which cannot beusually operated by touch.

FIG. 5 is a schematic diagram illustrating that a remote image object isoperated by switching the input mode. In FIG. 5, the cursor position P6is determined as the operation position on the screen (on the displaysurface) based on the position and direction of the finger (indexfinger) obtained from the position coordinate P4 of the fingertip andthe position coordinate P5 of the base of the finger. When the amount ofchange in the direction perpendicular to the display surface among theamount of change in position of a finger (fingertip) per a predeterminedtime period (280 msec, for example) is equal to or more than apredetermined threshold (13 cm, for example) in the direction close tothe display surface, the following processing is executed. Specifically,for example, the cursor position is stored at the point of timecorresponding to the starting point of the predetermined time periodduring which the amount of change being equal to or more than thepredetermined threshold is detected. The position is determined wherethe position at which the contact between finger and the display surfaceis detected, as illustrated in FIG. 5, the cursor position is movedaccording to user's operation for moving the contact position. Theoperation in which the cursor position is moved according to user'soperation for moving the contact position is executed based on thecontrol by the remote control unit 205.

In the present exemplary embodiment, the processing illustrated in FIG.4C may be executed instead of the processing in step S403 illustrated inFIG. 4B. In step S701, as is the case with step S601, the pointing inputprocessing unit 201 executes the processing for detecting a changingpoint from the relationship between the time T and the amount of changein the fingertip. If a predetermined amount of change in the fingertipis detected within a predetermined time period (YES in step S701), theprocessing in step S702 is performed. If a changing point is notdetected (NO in step S701), a series of processes is ended. In stepS702, the pointing input processing unit 201 determines whether a tap bythe user (contact between the digitizer and the hand) is detected. If atap is detected (YES in step S702), the processing in step S703 isperformed. If a tap is not detected (NO in step S702), a series ofprocesses is ended. In step S703, the pointing position is returned tothe position where the changing point is detected in step S701.

In the present exemplary embodiment, in step S402, the intersection ofthe straight line connecting the fingertip with the point of base of thefinger and the XY plane is set to a pointing position. In theembodiments, the intersection of the straight line connecting thefingertip with the point of center of gravity of a user's head, insteadof the base of the finger, and the XY plane maybe set to a pointingposition. In the present exemplary embodiment, the length of thedigitizer, a pointing position, a distance over which a finger isbrought down, and a time period are merely examples. Such length,position, distance, and time period may be appropriately set.

In the present exemplary embodiment, the projector is used to displayimage data, but a display unit such as a display may be used instead ofthe projector.

Other Embodiments

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiments, and by a method, the steps of whichare performed by a computer of a system or apparatus by, for example,reading out and executing a program recorded on a memory device toperform the functions of the above-described embodiments. For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (e.g., computer-readable medium). In such a case, thesystem or apparatus, and the recording medium where the program isstored, are included as being within the scope of the present invention.In an example, a computer-readable medium may store a program thatcauses an information processing apparatus to perform a method describedherein. In another example, a central processing unit (CPU) may beconfigured to control at least one unit utilized in a method orapparatus described herein.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all modifications, equivalent structures, and functions.

This application claims priority from Japanese Patent Application No.2010-188485 filed Aug. 25, 2010 and Japanese Patent Application No.2011-156692 filed Jul. 15, 2011, which are hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An information processing apparatus comprising: adetection unit configured to detect an amount of change in a position ofan object of interest per a predetermined time period; and a switchingunit configured to switch between a first mode for determining a firstoperation position on a display surface based on the position anddirection of an object of interest and a second mode for determining asecond operation position on the display surface based on a positionwhere the object of interest is in contact with the display surfaceusing the detected amount of change.
 2. The information processingapparatus according to claim 1, wherein the switching unit switchesbetween the first and second modes in a case where an amount of changein a direction perpendicular to the display surface per a predeterminedtime period among the amount of change in the position of the object ofinterest per the predetermined time period is equal to or more than apredetermined threshold.
 3. The information processing apparatusaccording to claim 1, further comprising: a change unit configured totake the first operation position in a predetermined point of timeincluded in a predetermined time period during which an amount of changein a perpendicular direction is equal to or more than a predeterminedthreshold as the first operation position in a case where the amount ofchange in the direction perpendicular to the display surface per thepredetermined time period among the amount of change in the position ofthe object of interest per the predetermined time period is equal to ormore than the predetermined threshold in a direction close to thedisplay surface in the first mode; and a control unit configured toexecute processing for the second operation position according to auser's operation to the first operation position in a case where thefirst mode is switched to the second mode.
 4. An information processingmethod comprising: detecting an amount of change in a position of anobject of interest per a predetermined time period; and switchingbetween a first mode for determining a first operation position on adisplay surface based on a position and direction of an object ofinterest and a second mode for determining a second operation positionon the display surface based on a position where the object of interestis in contact with the display surface using the detected amount ofchange.
 5. The information processing method according to claim 4,wherein switching includes switching between the first and second modesin a case where an amount of change in a direction perpendicular to thedisplay surface per a predetermined time period among the amount ofchange in the position of the object of interest per the predeterminedtime period is equal to or more than a predetermined threshold.
 6. Theinformation processing method according to claim 4, further comprising:taking the first operation position in a predetermined point of timeincluded in a predetermined time period during which an amount of changein a perpendicular direction is equal to or more than a predeterminedthreshold as the first operation position in a case where the amount ofchange in the direction perpendicular to the display surface per thepredetermined time period among the amount of change in the position ofthe object of interest per the predetermined time period is equal to ormore than the predetermined threshold in a direction close to thedisplay surface in the first mode; and executing processing for thesecond operation position according to a user's operation to the firstoperation position in a case where the first mode is switched to thesecond mode.
 7. A non-transitory computer-readable medium storing aprogram causing an information processing apparatus to perform a methodaccording to claim
 6. 8. A computer storing a computer program forcausing the computer to function as units of an information processingapparatus according to claim 1.